{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d717e88a",
   "metadata": {},
   "source": [
    "### 添加n-gram特征"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1202d4c6",
   "metadata": {},
   "source": [
    "n-gram特征：给定一段文本序列，其中n个词或字的相邻共现特征即n-gram特征，常用的n-gram特征是bi-gram和tri-gram特征，分别对应n为2和3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3448727f",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_gram = 2\n",
    "\n",
    "def create_ngram_set(input_list):\n",
    "    return set(zip(*[input_list[i: ] for i in range(n_gram)]))\n",
    "\n",
    "input_list = [1,3,2,1,5,3]\n",
    "res = create_ngram_set(input_list)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e7634a",
   "metadata": {},
   "source": [
    "### 文本长度规范"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc811bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing import sequence\n",
    "\n",
    "# 最好能覆盖90%左右的最短长度\n",
    "cutlen = 10\n",
    "\n",
    "def padding(x_train):\n",
    "    '''\n",
    "    x_train: 文本的张量表示，形如：[[1,32,32,61], [2,54,21,7,19]]\n",
    "    '''\n",
    "    return sequence.pad_sequences(x_train, cutlen)\n",
    "\n",
    "x_train = [\n",
    "    [1,23,5,32,55,63,2,21,78,32,23,1],\n",
    "    [2,32,1,23,1]\n",
    "]\n",
    "res = padding(x_train)\n",
    "res "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5af1739",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torchX",
   "language": "python",
   "name": "torchx"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
